Inferring State of Traffic Signal and Other Aspects of a Vehicle&#39;s Environment Based on Surrogate Data

ABSTRACT

A vehicle configured to operate in an autonomous mode can obtain sensor data from one or more sensors observing one or more aspects of an environment of the vehicle. At least one aspect of the environment of the vehicle that is not observed by the one or more sensors could be inferred based on the sensor data. The vehicle could be controlled in the autonomous mode based on the at least one inferred aspect of the environment of the vehicle.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of Ser. No. 15/087,390, filed Mar.31, 2016, which is a continuation of application Ser. No. 14/308,409,filed Jun. 18, 2014, which is a continuation of application Ser. No.13/486,886, filed Jun. 1, 2012. The prior applications are incorporatedherein by reference.

BACKGROUND

Unless otherwise indicated herein, the materials described in thissection are not prior art to the claims in this application and are notadmitted to be prior art by inclusion in this section.

Some vehicles are configured to operate in an autonomous mode in whichthe vehicle navigates through an environment with little or no inputfrom a driver. Such a vehicle typically includes one or more sensorsthat are configured to sense information about the environment. Thevehicle may use the sensed information to navigate through theenvironment. For example, if the sensors sense that the vehicle isapproaching an obstacle, the vehicle may navigate around the obstacle.

SUMMARY

In a first aspect, a method is provided. The method includes obtainingsensor data from one or more sensors observing one or more aspects of anenvironment of a vehicle. The one or more sensors are operationallyassociated with the vehicle. The vehicle is configured to operate in anautonomous mode. The method further includes using an inference systemto infer, based on the sensor data, at least one aspect of theenvironment of the vehicle that is not observed by the one or moresensors. The method also includes controlling the vehicle in theautonomous mode based on the at least one inferred aspect of theenvironment of the vehicle.

In a second aspect, a vehicle is provided. The vehicle includes one ormore sensors, an inference system, and a control system. The one or moresensors are configured to acquire sensor data. The sensor data relatesto one or more aspects of an environment of a vehicle observed by theone or more sensors. The vehicle is configured to operate in anautonomous mode. The inference system is configured to infer, based onthe sensor data, at least one aspect of the environment of the vehiclethat is not observed by the one or more sensors. The control system isconfigured to control the vehicle in the autonomous mode based on the atleast one inferred aspect of the environment of the vehicle.

In a third aspect, a non-transitory computer readable medium havingstored instructions is provided. The instructions are executable by acomputer system to cause the computer system to perform functions. Thefunctions include obtaining sensor data from one or more sensorsobserving one or more aspects of an environment of a vehicle. The one ormore sensors are operationally associated with the vehicle. The vehicleis configured to operate in an autonomous mode. The functions furtherinclude inferring, based on the sensor data, at least one aspect of theenvironment of the vehicle that is not observed by the one or moresensors. The functions also include controlling the vehicle in theautonomous mode based on the at least one inferred aspect of theenvironment of the vehicle.

In a fourth aspect, a method is provided. The method includes obtainingsensor data from one or more sensors observing at least one light sourcein an environment of the vehicle. The one or more sensors areoperationally associated with the vehicle and a state of a controllingtraffic signal is not directly observable using the one or more sensors.The vehicle is configured to operate in an autonomous mode. The methodfurther includes using an inference system to infer, based on the sensordata, an inferred state of the controlling traffic signal. The methodalso includes controlling the vehicle in the autonomous mode based onthe inferred state of the controlling traffic signal.

The foregoing summary is illustrative only and is not intended to be inany way limiting. In addition to the illustrative aspects, embodiments,and features described above, further aspects, embodiments, and featureswill become apparent by reference to the figures and the followingdetailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional block diagram illustrating a vehicle, accordingto an example embodiment.

FIG. 2 shows a vehicle, according to an example embodiment.

FIG. 3A is a top view of an autonomous vehicle operating scenario,according to an example embodiment.

FIG. 3B is a rear view of the autonomous vehicle operating scenario ofFIG. 3A, according to an example embodiment.

FIG. 3C is a top view of an autonomous vehicle operating scenario,according to an example embodiment.

FIG. 3D is a top view of an autonomous vehicle operating scenario,according to an example embodiment.

FIG. 4A shows a method, according to an example embodiment.

FIG. 4B shows a method, according to an example embodiment.

FIG. 5 is a schematic diagram of a computer program product, accordingto an example embodiment.

DETAILED DESCRIPTION

Example methods and systems are described herein. Any example embodimentor feature described herein is not necessarily to be construed aspreferred or advantageous over other embodiments or features. Theexample embodiments described herein are not meant to be limiting. Itwill be readily understood that certain aspects of the disclosed systemsand methods can be arranged and combined in a wide variety of differentconfigurations, all of which are contemplated herein.

Furthermore, the particular arrangements shown in the Figures should notbe viewed as limiting. It should be understood that other embodimentsmay include more or less of each element shown in a given Figure.Further, some of the illustrated elements may be combined or omitted.Yet further, an example embodiment may include elements that are notillustrated in the Figures.

1. Overview

Example embodiments disclosed herein relate to obtaining sensor datafrom one or more sensors observing one or more aspects of an environmentof a vehicle, using an inference system to infer, based on the sensordata, at least one aspect of the environment of the vehicle that is notobserved by the one or more sensors, and controlling the vehicle basedon the at least one inferred aspect of the environment of the vehicle.

Within the context of the disclosure, the vehicle could be operable invarious modes of operation. In some embodiments, such modes of operationcould include manual, semi-autonomous, and autonomous modes. In theautonomous mode, the vehicle could be driven with little or no userinteraction. In the manual and semi-autonomous modes, the vehicle couldbe driven entirely and partially, respectively, by a user.

Some methods disclosed herein could be carried out in part or in full bya vehicle configured to operate in an autonomous mode with or withoutexternal interaction (e.g., such as from a user of the vehicle). In onesuch example, a vehicle could obtain sensor data from one or moresensors operationally associated with the vehicle. The sensors could belocated on-board the vehicle or the sensors could be elsewhere. Aninference system, which could be located fully or partially on-board thevehicle, may be used to infer, based on the sensor data, at least oneaspect of the environment of the vehicle that is not observed by the oneor more sensors. In an example embodiment, the at least one aspect ofthe environment of the vehicle that is not observed by the one or moresensors could be related to traffic flow (e.g., an unobservable accidentahead) and/or traffic regulation (e.g., state of an unobservable trafficlight, order of vehicles proceeding at a four-way stop, etc.). Otheraspects of the environment of the vehicles that are not observed by theone or more sensors are possible. The inference system could include,for example, a computer system (e.g., a processor and a memory). Thevehicle could be controlled based on the at least one inferred aspect ofthe environment of the vehicle.

Other methods disclosed herein could be carried out in part or in fullby a server. In example embodiment, a server may receive sensor datafrom one or more sensors observing one or more aspects of an environmentof a vehicle. In some embodiments, the sensor data could be transmittedto the server using a wireless communication system. The server mayinclude an inference system. The inference system could be used toinfer, based on the sensor data, at least one aspect of the environmentof the vehicle that is not observed by the one or more sensors. In anexample embodiment, the server could include a data store ofpredetermined scenarios. If the sensor data substantially matches atleast one of the predetermined scenarios, the inference system couldinfer the at least one aspect of the environment of the vehicle that isnot observed by the one or more sensors. Other interactions between avehicle operating in an autonomous mode and a server are possible withinthe context of the disclosure.

A vehicle is also described in the present disclosure. The vehicle mayinclude elements such as one or more sensors, an inference system, and acontrol system. The one or more sensors are configured to acquire sensordata. The sensor data relates to one or more aspects of an environmentof a vehicle observed by the one or more sensors.

The one or more sensors could include one of, or a combination of, acamera, a RADAR system, a LIDAR system, an acoustic sensor, arangefinder, or another type of sensor.

The inference system could be configured to infer, based on the sensordata, at least one aspect of the environment of the vehicle that is notobserved by the one or more sensors. The inference system could beconfigured to use one or more algorithms in order to determine theinference. The algorithms could include, for example, one or more of aBayesian network, a hidden Markov model, and a decision tree. Othertypes of inference systems are possible within the context of thisdisclosure.

The control system could be configured to control the vehicle in theautonomous mode based on the at least one inferred aspect of theenvironment of the vehicle. The control system could be operable tocontrol the vehicle to speed up, slow down, adjust heading, reroute, andtake evasive action, among many other possibilities.

In some embodiments, the inference system and the control system couldbe provided by a computer system in the vehicle. In other embodiments,the inference system and/or the control system could be provided by oneor more servers or other computer systems external to the vehicle.

Also disclosed herein are non-transitory computer readable media withstored instructions. The instructions could be executable by a computingdevice to cause the computing device to perform functions similar tothose described in the aforementioned methods.

There are many different specific methods and systems that could be usedin obtaining sensor data from one or more sensors observing one or moreaspects of an environment of a vehicle. An inference system could beused to infer, based on the sensor data, at least one aspect of theenvironment of the vehicle that is not observed by the one or moresensors. The vehicle could be controlled based on the at least oneinferred aspect of the environment of the vehicle. Each of thesespecific methods and systems are contemplated herein, and severalexample embodiments are described below.

2. Example Systems

Example systems within the scope of the present disclosure will now bedescribed in greater detail. Generally, an example system may beimplemented in or may take the form of a vehicle. However, an examplesystem may also be implemented in or take the form of other vehicles,such as cars, trucks, motorcycles, buses, boats, airplanes, helicopters,lawn mowers, earth movers, boats, snowmobiles, aircraft, recreationalvehicles, amusement park vehicles, farm equipment, constructionequipment, trams, golf carts, trains, and trolleys. Other vehicles arepossible as well.

FIG. 1 is a functional block diagram illustrating a vehicle 100,according to an example embodiment. The vehicle 100 could be configuredto operate fully or partially in an autonomous mode. For example, thevehicle 100 could control itself while in the autonomous mode, and maybe operable to determine a current state of the vehicle and itsenvironment, determine a predicted behavior of at least one othervehicle in the environment, determine a confidence level that maycorrespond to a likelihood of the at least one other vehicle to performthe predicted behavior, and control the vehicle 100 based on thedetermined information. While in autonomous mode, the vehicle 100 may beconfigured to operate without human interaction.

The vehicle 100 could include various subsystems such as a propulsionsystem 102, a sensor system 104, a control system 106, one or moreperipherals 108, as well as a power supply 110, a computer system 112,and a user interface 116. The vehicle 100 may include more or fewersubsystems and each subsystem could include multiple elements. Further,each of the subsystems and elements of vehicle 100 could beinterconnected. Thus, one or more of the described functions of thevehicle 100 may be divided up into additional functional or physicalcomponents, or combined into fewer functional or physical components. Insome further examples, additional functional and/or physical componentsmay be added to the examples illustrated by FIG. 1.

The propulsion system 102 may include components operable to providepowered motion for the vehicle 100. In an example embodiment, thepropulsion system 102 could include an engine/motor 118, an energysource 119, a transmission 120, and wheels/tires 121. The engine/motor118 could be any combination of an internal combustion engine, anelectric motor, steam engine, Stirling engine, or other types of enginesand/or motors. In some embodiments, the engine/motor 118 may beconfigured to convert energy source 119 into mechanical energy. In someembodiments, the propulsion system 102 could include multiple types ofengines and/or motors. For instance, a gas-electric hybrid car couldinclude a gasoline engine and an electric motor. Other examples arepossible.

The energy source 119 could represent a source of energy that may, infull or in part, power the engine/motor 118. That is, the engine/motor118 could be configured to convert the energy source 119 into mechanicalenergy. Examples of energy sources 119 include gasoline, diesel, otherpetroleum-based fuels, propane, other compressed gas-based fuels,ethanol, solar panels, batteries, and other sources of electrical power.The energy source(s) 119 could additionally or alternatively include anycombination of fuel tanks, batteries, capacitors, and/or flywheels. Theenergy source 119 could also provide energy for other systems of thevehicle 100.

The transmission 120 could include elements that are operable totransmit mechanical power from the engine/motor 118 to the wheels/tires121. To this end, the transmission 120 could include a gearbox, clutch,differential, and drive shafts. The transmission 120 could include otherelements. The drive shafts could include one or more axles that could becoupled to the one or more wheels/tires 121.

The wheels/tires 121 of vehicle 100 could be configured in variousformats, including a unicycle, bicycle/motorcycle, tricycle, orcar/truck four-wheel format. Other wheel/tire geometries are possible,such as those including six or more wheels. Any combination of thewheels/tires 121 of vehicle 100 may be operable to rotate differentiallywith respect to other wheels/tires 121. The wheels/tires 121 couldrepresent at least one wheel that is fixedly attached to thetransmission 120 and at least one tire coupled to a rim of the wheelthat could make contact with the driving surface. The wheels/tires 121could include any combination of metal and rubber, or anothercombination of materials.

The sensor system 104 may include a number of sensors configured tosense information about an environment of the vehicle 100. For example,the sensor system 104 could include a Global Positioning System (GPS)122, an inertial measurement unit (IMU) 124, a RADAR unit 126, a laserrangefinder/LIDAR unit 128, and a camera 130. The sensor system 104could also include sensors configured to monitor internal systems of thevehicle 100 (e.g., O₂ monitor, fuel gauge, engine oil temperature).Other sensors are possible as well.

One or more of the sensors included in sensor system 104 could beconfigured to be actuated separately and/or collectively in order tomodify a position and/or an orientation of the one or more sensors.

The GPS 122 may be any sensor configured to estimate a geographiclocation of the vehicle 100. To this end, GPS 122 could include atransceiver operable to provide information regarding the position ofthe vehicle 100 with respect to the Earth.

The IMU 124 could include any combination of sensors (e.g.,accelerometers and gyroscopes) configured to sense position andorientation changes of the vehicle 100 based on inertial acceleration.

The RADAR unit 126 may represent a system that utilizes radio signals tosense objects within the local environment of the vehicle 100. In someembodiments, in addition to sensing the objects, the RADAR unit 126 mayadditionally be configured to sense the speed and/or heading of theobjects.

Similarly, the laser rangefinder or LIDAR unit 128 may be any sensorconfigured to sense objects in the environment in which the vehicle 100is located using lasers. In an example embodiment, the laserrangefinder/LIDAR unit 128 could include one or more laser sources, alaser scanner, and one or more detectors, among other system components.The laser rangefinder/LIDAR unit 128 could be configured to operate in acoherent (e.g., using heterodyne detection) or an incoherent detectionmode.

The camera 130 could include one or more devices configured to capture aplurality of images of the environment of the vehicle 100. The camera130 could be a still camera or a video camera.

The control system 106 may be configured to control operation of thevehicle 100 and its components. Accordingly, the control system 106could include various elements include steering unit 132, throttle 134,brake unit 136, a sensor fusion algorithm 138, a computer vision system140, a navigation/pathing system 142, and an obstacle avoidance system144.

The steering unit 132 could represent any combination of mechanisms thatmay be operable to adjust the heading of vehicle 100.

The throttle 134 could be configured to control, for instance, theoperating speed of the engine/motor 118 and, in turn, control the speedof the vehicle 100.

The brake unit 136 could include any combination of mechanismsconfigured to decelerate the vehicle 100. The brake unit 136 could usefriction to slow the wheels/tires 121. In other embodiments, the brakeunit 136 could convert the kinetic energy of the wheels/tires 121 toelectric current. The brake unit 136 may take other forms as well.

The sensor fusion algorithm 138 may be an algorithm (or a computerprogram product storing an algorithm) configured to accept data from thesensor system 104 as an input. The data may include, for example, datarepresenting information sensed at the sensors of the sensor system 104.The sensor fusion algorithm 138 could include, for instance, a Kalmanfilter, Bayesian network, or other algorithm. The sensor fusionalgorithm 138 could further provide various assessments based on thedata from sensor system 104. In an example embodiment, the assessmentscould include evaluations of individual objects and/or features in theenvironment of vehicle 100, evaluation of a particular situation, and/orevaluate possible impacts based on the particular situation. Otherassessments are possible.

The computer vision system 140 may be any system operable to process andanalyze images captured by camera 130 in order to identify objectsand/or features in the environment of vehicle 100 that could includetraffic signals, road way boundaries, and obstacles. The computer visionsystem 140 could use an object recognition algorithm, a Structure FromMotion (SFM) algorithm, video tracking, and other computer visiontechniques. In some embodiments, the computer vision system 140 could beadditionally configured to map an environment, track objects, estimatethe speed of objects, etc.

The navigation and pathing system 142 may be any system configured todetermine a driving path for the vehicle 100. The navigation and pathingsystem 142 may additionally be configured to update the driving pathdynamically while the vehicle 100 is in operation. In some embodiments,the navigation and pathing system 142 could be configured to incorporatedata from the sensor fusion algorithm 138, the GPS 122, and one or morepredetermined maps so as to determine the driving path for vehicle 100.

The obstacle avoidance system 144 could represent a control systemconfigured to identify, evaluate, and avoid or otherwise negotiatepotential obstacles in the environment of the vehicle 100.

The control system 106 may additionally or alternatively includecomponents other than those shown and described.

Peripherals 108 may be configured to allow interaction between thevehicle 100 and external sensors, other vehicles, other computersystems, and/or a user. For example, peripherals 108 could include awireless communication system 146, a touchscreen 148, a microphone 150,and/or a speaker 152.

In an example embodiment, the peripherals 108 could provide, forinstance, means for a user of the vehicle 100 to interact with the userinterface 116. To this end, the touchscreen 148 could provideinformation to a user of vehicle 100. The user interface 116 could alsobe operable to accept input from the user via the touchscreen 148. Thetouchscreen 148 may be configured to sense at least one of a positionand a movement of a user's finger via capacitive sensing, resistancesensing, or a surface acoustic wave process, among other possibilities.The touchscreen 148 may be capable of sensing finger movement in adirection parallel or planar to the touchscreen surface, in a directionnormal to the touchscreen surface, or both, and may also be capable ofsensing a level of pressure applied to the touchscreen surface. Thetouchscreen 148 may be formed of one or more translucent or transparentinsulating layers and one or more translucent or transparent conductinglayers. The touchscreen 148 may take other forms as well.

In other instances, the peripherals 108 may provide means for thevehicle 100 to communicate with devices within its environment. Themicrophone 150 may be configured to receive audio (e.g., a voice commandor other audio input) from a user of the vehicle 100. Similarly, thespeakers 152 may be configured to output audio to the user of thevehicle 100.

In one example, the wireless communication system 146 could beconfigured to wirelessly communicate with one or more devices directlyor via a communication network. For example, wireless communicationsystem 146 could use 3G cellular communication, such as CDMA, EVDO,GSM/GPRS, or 4G cellular communication, such as WiMAX or LTE.Alternatively, wireless communication system 146 could communicate witha wireless local area network (WLAN), for example, using WiFi. In someembodiments, wireless communication system 146 could communicatedirectly with a device, for example, using an infrared link, Bluetooth,or ZigBee. Other wireless protocols, such as various vehicularcommunication systems, are possible within the context of thedisclosure. For example, the wireless communication system 146 couldinclude one or more dedicated short range communications (DSRC) devicesthat could include public and/or private data communications betweenvehicles and/or roadside stations.

The power supply 110 may provide power to various components of vehicle100 and could represent, for example, a rechargeable lithium-ion orlead-acid battery. In some embodiments, one or more banks of suchbatteries could be configured to provide electrical power. Other powersupply materials and configurations are possible. In some embodiments,the power supply 110 and energy source 119 could be implementedtogether, as in some all-electric cars.

Many or all of the functions of vehicle 100 could be controlled bycomputer system 112. Computer system 112 may include at least oneprocessor 113 (which could include at least one microprocessor) thatexecutes instructions 115 stored in a non-transitory computer readablemedium, such as the data storage 114. The computer system 112 may alsorepresent a plurality of computing devices that may serve to controlindividual components or subsystems of the vehicle 100 in a distributedfashion.

In some embodiments, data storage 114 may contain instructions 115(e.g., program logic) executable by the processor 113 to execute variousfunctions of vehicle 100, including those described above in connectionwith FIG. 1. Data storage 114 may contain additional instructions aswell, including instructions to transmit data to, receive data from,interact with, and/or control one or more of the propulsion system 102,the sensor system 104, the control system 106, and the peripherals 108.

In addition to the instructions 115, the data storage 114 may store datasuch as roadway maps, path information, among other information. Suchinformation may be used by vehicle 100 and computer system 112 at duringthe operation of the vehicle 100 in the autonomous, semi-autonomous,and/or manual modes.

The vehicle 100 may include a user interface 116 for providinginformation to or receiving input from a user of vehicle 100. The userinterface 116 could control or enable control of content and/or thelayout of interactive images that could be displayed on the touchscreen148. Further, the user interface 116 could include one or moreinput/output devices within the set of peripherals 108, such as thewireless communication system 146, the touchscreen 148, the microphone150, and the speaker 152.

The computer system 112 may control the function of the vehicle 100based on inputs received from various subsystems (e.g., propulsionsystem 102, sensor system 104, and control system 106), as well as fromthe user interface 116. For example, the computer system 112 may utilizeinput from the control system 106 in order to control the steering unit132 to avoid an obstacle detected by the sensor system 104 and theobstacle avoidance system 144. In an example embodiment, the computersystem 112 could be operable to provide control over many aspects of thevehicle 100 and its subsystems.

Although FIG. 1 shows various components of vehicle 100, i.e., wirelesscommunication system 146, computer system 112, data storage 114, anduser interface 116, as being integrated into the vehicle 100, one ormore of these components could be mounted or associated separately fromthe vehicle 100. For example, data storage 114 could, in part or infull, exist separate from the vehicle 100. Thus, the vehicle 100 couldbe provided in the form of device elements that may be locatedseparately or together. The device elements that make up vehicle 100could be communicatively coupled together in a wired and/or wirelessfashion.

FIG. 2 shows a vehicle 200 that could be similar or identical to vehicle100 described in reference to FIG. 1. Although vehicle 200 isillustrated in FIG. 2 as a car, other embodiments are possible. Forinstance, the vehicle 200 could represent a truck, a van, a semi-trailertruck, a motorcycle, a golf cart, an off-road vehicle, or a farmvehicle, among other examples.

In some embodiments, vehicle 200 could include a sensor unit 202, awireless communication system 204, a LIDAR unit 206, a laser rangefinderunit 208, and a camera 210. The elements of vehicle 200 could includesome or all of the elements described for FIG. 1.

The sensor unit 202 could include one or more different sensorsconfigured to capture information about an environment of the vehicle200. For example, sensor unit 202 could include any combination ofcameras, RADARs, LIDARs, range finders, and acoustic sensors. Othertypes of sensors are possible. In an example embodiment, the sensor unit202 could include one or more movable mounts that could be operable toadjust the orientation of one or more sensors in the sensor unit 202. Inone embodiment, the movable mount could include a rotating platform thatcould scan sensors so as to obtain information from each directionaround the vehicle 200. In another embodiment, the movable mount of thesensor unit 202 could be moveable in a scanning fashion within aparticular range of angles and/or azimuths. The sensor unit 202 could bemounted atop the roof of a car, for instance, however other mountinglocations are possible. Additionally, the sensors of sensor unit 202could be distributed in different locations and need not be collocatedin a single location. Some possible sensor types and mounting locationsinclude LIDAR unit 206 and laser rangefinder unit 208. Furthermore, eachsensor of sensor unit 202 could be configured to be moved or scannedindependently of other sensors of sensor unit 202.

The wireless communication system 204 could be located on a roof of thevehicle 200 as depicted in FIG. 2. Alternatively, the wirelesscommunication system 204 could be located, fully or in part, elsewhere.The wireless communication system 204 may include wireless transmittersand receivers that could be configured to communicate with devicesexternal or internal to the vehicle 200. Specifically, the wirelesscommunication system 204 could include transceivers configured tocommunicate with other vehicles and/or computing devices, for instance,in a vehicular communication system or a roadway station. Examples ofsuch vehicular communication systems include dedicated short rangecommunications (DSRC), radio frequency identification (RFID), and otherproposed communication standards directed towards intelligent transportsystems.

The camera 210 may be any camera (e.g., a still camera, a video camera,etc.) configured to capture a plurality of images of the environment ofthe vehicle 200. To this end, the camera 210 may be configured to detectvisible light, or may be configured to detect light from other portionsof the spectrum, such as infrared or ultraviolet light. Other types ofcameras are possible as well.

The camera 210 may be a two-dimensional detector, or may have athree-dimensional spatial range. In some embodiments, the camera 210 maybe, for example, a range detector configured to generate atwo-dimensional image indicating a distance from the camera 210 to anumber of points in the environment. To this end, the camera 210 may useone or more range detecting techniques. For example, the camera 210 mayuse a structured light technique in which the vehicle 200 illuminates anobject in the environment with a predetermined light pattern, such as agrid or checkerboard pattern and uses the camera 210 to detect areflection of the predetermined light pattern off the object. Based ondistortions in the reflected light pattern, the vehicle 200 maydetermine the distance to the points on the object. The predeterminedlight pattern may comprise infrared light, or light of anotherwavelength. As another example, the camera 210 may use a laser scanningtechnique in which the vehicle 200 emits a laser and scans across anumber of points on an object in the environment. While scanning theobject, the vehicle 200 uses the camera 210 to detect a reflection ofthe laser off the object for each point. Based on a length of time ittakes the laser to reflect off the object at each point, the vehicle 200may determine the distance to the points on the object. As yet anotherexample, the camera 210 may use a time-of-flight technique in which thevehicle 200 emits a light pulse and uses the camera 210 to detect areflection of the light pulse off an object at a number of points on theobject. In particular, the camera 210 may include a number of pixels,and each pixel may detect the reflection of the light pulse from a pointon the object. Based on a length of time it takes the light pulse toreflect off the object at each point, the vehicle 200 may determine thedistance to the points on the object. The light pulse may be a laserpulse. Other range detecting techniques are possible as well, includingstereo triangulation, sheet-of-light triangulation, interferometry, andcoded aperture techniques, among others. The camera 210 may take otherforms as well.

The camera 210 could be mounted inside a front windshield of the vehicle200. Specifically, as illustrated, the camera 210 could capture imagesfrom a forward-looking view with respect to the vehicle 200. Othermounting locations and viewing angles of camera 210 are possible, eitherinside or outside the vehicle 200.

The camera 210 could have associated optics that could be operable toprovide an adjustable field of view. Further, the camera 210 could bemounted to vehicle 200 with a movable mount that could be operable tovary a pointing angle of the camera 210.

Within the context of the present disclosure, the components of vehicle100 and/or vehicle 200 could be configured to work in an interconnectedfashion with other components within or outside their respectivesystems. For instance, in an example embodiment, the camera 130 couldcapture a plurality of images that could represent sensor data relatingto an environment of the vehicle 100 operating in an autonomous mode.The environment could include another vehicle blocking a known trafficsignal location ahead of the vehicle 100. Based on the plurality ofimages, an inference system (which could include the computer system112, sensor system 104, and control system 106) could infer that theunobservable traffic signal is red based on sensor data from otheraspects of the environment (for instance images indicating the blockingvehicle's brake lights are on). Based on the inference, the computersystem 112 and propulsion system 102 could act to control the vehicle100. More detailed example implementations will be discussed below.

3. Example Implementations

Several example implementations will now be described herein. It will beunderstood that there are many ways to implement the devices, systems,and methods disclosed herein. Accordingly, the following examples arenot intended to limit the scope of the present disclosure.

FIG. 3A and FIG. 3B illustrate a top view and a back view, respectively,of an autonomous vehicle operating scenario 300. In scenario 300, avehicle 306, which may be operating in an autonomous mode, is stoppedbehind a large truck 304. The vehicle 306 could be similar or identicalto the vehicle 100 and vehicle 200 as described in reference to FIG. 1and FIG. 2. The vehicle 306 could include a sensor system 310. The largetruck 304 could be obscuring a traffic signal 302 such that the sensorsystem 310 may not be able to directly sense the state of the trafficsignal 302. In other words, in scenario 300, none of the sensors insensor system 310 may be able to observe, image, transduce or otherwisedetect the state of the traffic signal 302 due to the presence of largetruck 304.

In such a case, sensor system 310 could be operable to obtain sensordata regarding other aspects of an environment of vehicle 306 in aneffort to determine the state of the traffic signal 302 or any otherrelevant unobservable aspect of its environment. In an exampleembodiment, a camera associated with sensor system 310 could capture aplurality of images. The plurality of images could include informationabout one or more brake lights 308 of the large truck 304. The pluralityof images could be an input to an inference system.

The inference system could determine from the images that the one ormore brake lights 308 of the large truck 304 are lit. The inferencesystem could also include other sensor data that may confirm that thelarge truck 304 is stopped. For instance, the inference system couldobtain information about the environment of the vehicle 306 from RADARand/or LIDAR sensor systems.

The inference system could compare the sensor data to a set ofpredetermined scenarios in an effort to match the present sensor datawith a particular predetermined scenario. The predetermined scenarioscould be saved in data storage on-board the vehicle. Alternatively, thepredetermined scenarios could be stored elsewhere, such as in a servernetwork.

If a substantial match exists between a particular predeterminedscenario and the present sensor data, an inference could be maderegarding a possible state of the unseen traffic signal 302. Forinstance, in scenario 300, the inference could be that the state oftraffic signal 302 is “red” (because large truck 304 is stopped due toobeying the traffic signal 302).

Within the context of the disclosure, a substantial match could be madebased on any one of, or a combination of, a similar arrangement ofstopped vehicles, a similar roadway, a similar location/intersection, asimilar route, a similar time of day, or any other similar aspect of theenvironment of vehicle 306.

Inferences could be made by the inference system based on otheralgorithms. For instance, the inference could be made based on aBayesian network, a hidden Markov model, and/or a decision tree. Otheralgorithms could be used individually or in combination in order toinfer about an aspect of the environment.

Based on the inferred aspect of the environment of the vehicle 306, thevehicle 306 may be controlled to perform actions. In scenario 300, theaction could constitute staying stopped until the large truck 304 beginsto move again.

In some scenarios, one or more subsequent inferences could be made bythe inference system if conditions in the environment change, or ifconditions in the environment stay the same. For example, in scenario300, if the vehicle 306 stays stopped behind the large truck 304 forlonger than an expected traffic signal delay time, the vehicle 306 couldmake a subsequent inference that the large truck 304 is stopped due to abreak down and not obeying the traffic signal 302. In response to thesubsequent inference, the vehicle 306 may take action, for instance, tomove into an adjacent lane so as to move around the large truck 304.Subsequent inferences could be made on an on-going basis, at specificpredetermined intervals, or upon sensing specific triggering events,such as scenario-specific sensor data. Other subsequent inferences maybe possible within the context of this disclosure.

FIG. 3C illustrates a top view of an autonomous vehicle operatingscenario 320. Similar to scenario 300, a large truck 324 may be blockinga view of a north-bound traffic signal 326 for vehicle 322, which may beoperating in an autonomous mode. As such, sensor system 323 of vehicle322 may be impeded from directly observing or otherwise sensing thestate of the north-bound traffic signal 326. The vehicle 322 could besimilar or identical to the vehicle 100 and vehicle 200 as described inreference to FIG. 1 and FIG. 2.

In an effort to infer the state of the north-bound traffic signal 326,the vehicle 322 could use sensor data from observable aspects of theenvironment. For instance, sensor system 323 could use one or acombination of its associated sensors (e.g., a camera, a RADAR, a LIDAR,an acoustic sensor, an ultrasonic sensor, etc.) to observe other trafficor non-traffic indicators that could be used as inputs to the inferencesystem.

In scenario 320, one or more sensors associated with sensor system 323could observe car 328 turning right onto a south-bound roadway and truck332 proceeding on a west-bound roadway. The sensor data related toactions of the car 328 and the truck 332 could be used as an input tothe inference system. Based on the sensor data, the inference systemcould infer that the state of the traffic signal 326 is “red” for north-and south-bound traffic, and the state of the traffic signal 334 is“green” for east- and west-bound traffic. As described above, theinference system could compare the sensor data and other data to a setof predetermined scenarios. If a substantial match exists between thesensor data/other data and a specific predetermined scenario, aninference could be made based on the most likely outcome within thecontext of the specific predetermined scenario.

Based on the inference, for instance that the north-bound traffic signal326 is “red”, the vehicle 322 could be controlled to take action in theform of remaining stopped behind large truck 324.

FIG. 3D illustrates a top view of an autonomous vehicle operatingscenario 340. In scenario 340, a vehicle 342 could be stopped at afour-way stop intersection, obeying north-bound stop sign 344. Thevehicle 342 could be operating in an autonomous mode. The vehicle 342could be similar or identical to the vehicle 100 and vehicle 200 asdescribed in reference to FIG. 1 and FIG. 2. Truck 346 may also bestopped at the intersection due to east-bound stop sign 348. In such ascenario, the vehicle 342 may want to determine a state of a trafficprogression of the four-way stop intersection. However, such a state oftraffic progression may not be observable. In other words, the vehicle342 may be uncertain about which vehicle should progress through theintersection next and there may be no directly observable signal (e.g.,a traffic signal) to indicate the progression order.

Sensor system 343 may be operable to sense other observable parts of theenvironment of vehicle 343 that could provide information about thestate of the traffic progression in the four-way stop intersection. Forexample, the sensor system 343 could use one or more associated sensorsto observe car 350 moving south-bound through the intersection. Thesensor data could be obtained by the inference system.

Based on the sensor data, the inference system could infer about thestate of the traffic progression in the four-way stop intersection. Inthis case, the inference system could determine that it is too late toprogress north-bound through the intersection and infer that the truck346 should move forward through the intersection next.

Based on the inference, the vehicle 342 could be controlled to wait fortruck 346 to clear the intersection before proceeding through theintersection.

4. Example Methods

A method 400 is provided for obtaining sensor data from one or moresensors observing one or more aspects of an environment of a vehicle.The one or more sensors are operationally associated with the vehicle.The method further includes using an inference system to infer, based onthe sensor data, at least one aspect of the environment of the vehiclethat is not observed by the one or more sensors and controlling thevehicle based on the at least one inferred aspect of the environment ofthe vehicle. The method could be performed using any of the apparatusshown in FIGS. 1 and 2 and described above, however, otherconfigurations could be used. FIG. 4A illustrates the steps in anexample method, however, it is understood that in other embodiments, thesteps may appear in different order and steps could be added orsubtracted.

Step 402 includes obtaining sensor data from one or more sensorsobserving one or more aspects of an environment of a vehicle. The one ormore sensors are operationally associated with the vehicle. The vehicleis configured to operate in an autonomous mode. The vehicle described inthis method could be the vehicle 100 and/or vehicle 200 as illustratedand described in reference to FIGS. 1 and 2, respectively.

Obtaining sensor data could include using one or more of a camera, aradar system, a lidar system, and an acoustic-sensing system. Othersensors and sensor systems are possible in the context of thedisclosure. The sensor data could be received by wired and/or wirelesssignals from the aforementioned sensors. In some embodiments, the sensordata could be obtained by a server network and communicated to thevehicle. Alternatively or additionally, the sensor data could beobtained by a computer system incorporated in the vehicle.

Step 404 includes using an inference system to infer, based on thesensor data, at least one aspect of the environment of the vehicle thatis not observed by the one or more sensors. The inference system couldbe provided by the computer system 112 as shown and described inreference to FIG. 1. In other embodiments, the inference system could bepartially or wholly provided by one or more computer systems external tothe vehicle, such as in a server network.

The inference system could be operable to make inferences about anynumber of aspects of the environment of the vehicle not directlyobserved by the one or more sensors associated with the vehicle. Forinstance, the inference system could make an inference about a state ofan unseen traffic signal or four-way stop. In other embodiments, theinference system could make an inference about a traffic density on aroadway, a speed limit, a road condition, a behavior of another vehicle,a road route, and/or an accident or obstacle impeding a roadway. Otheraspects of the environment could be inferred.

The inference system could make inferences based on sensor data from oneof, or any combination of, sensors operationally-associated with thevehicle. The sensors could be located on-board the vehicle.Alternatively, some of the sensors could be located apart from thevehicle.

The inferences made by the inference system could be based on any aspectof the environment of the vehicle observable to the sensors. In anexample embodiment, inferences could be made from sensor data regardingthe other vehicles. The sensor data could relate to, for instance, thespeed, heading, and/or position of other vehicles. For example,inferences could be made based on the acceleration and/or deceleration(with or without brake lights) of other vehicles when approaching anintersection with an occluded traffic signal. Deceleration of the othervehicles may imply a ‘red’ traffic signal, while acceleration may implya ‘green’ traffic signal. Other examples are possible with regards tousing sensor data based on other vehicles.

Inferences could also be made based on the sensed position of obstacles,changes in the observed roadway compared to a known map, condition ofthe roadway, etc. Further, sensor data from more than one sensor couldbe used by the inference system to make an inference.

The inference system could include artificial intelligence and/or alearning algorithm. In such cases, the inference system could inferabout an aspect of the environment of the vehicle based on obtainedsensor data. The sensor data could be compared against a set ofrepresentative sensor input scenarios or predetermined scenarios thatcould represent a set of similar scenarios with a set of possibleoutcomes. That is, for a given a set of sensor inputs, a set of possibleoutcomes could be associated to the given set of sensor inputs. Eachpossible outcome associated with the given set of sensor inputs could beassociated with a probability. The probability could relate to thepossible outcome actually occurring.

The inference made by the inference system could relate to a ‘bestguess’ for the outcome, which may include one of the possible outcomes.If the inference is confirmed (e.g., the ‘best guess’ was correct), thepossible outcome within the set of representative sensor input scenarioscould be reinforced by, for instance, increasing the probability thatthe possible outcome will occur. If the inference is determined to beincorrect, the probability could be decreased for the given possibleoutcome among the set of representative sensor input scenarios orpredetermined scenarios. Other types of learning algorithms arepossible.

Each predetermined scenario or inference could include an associatedconfidence level based on how confident the inference system may be inthe predicted outcome. The confidence level may or may not directlycorrelate with the respective probability of the most-likely outcome ineach predetermined scenario. The confidence level could be based on, forinstance, one or more actual outcomes following one or more previousinferences. In such an example, if a previous inference incorrectlypredicted the actual outcome, the confidence level of the predeterminedscenario could be subsequently decreased. Conversely, if a previousinference correctly predicted the actual outcome, the confidence levelof the predetermined scenario could be increased.

Within the context of the disclosure, inferences could be made using afail-safe approach. In other words, the inference system could makeinferences that may be correct or incorrect representations of anunobservable aspect of the environment. In a fail-safe approach,incorrect inferences (e.g., inferring that a traffic light is “red” whenit is actually “green”) could be carried out within the context of safevehicle operation so as to avoid unsafe operation.

For example, various traffic rules could be used or modified by theinference system in an effort to provide fail-safe operation. An actualtraffic rule could be: ‘Unless you know a traffic signal is green, donot proceed through the intersection.’ The inference system could use amodified version of the rule such as: ‘Unless you directly observe atraffic signal as being green or unless other vehicles in your lane areproceeding through the intersection, do not proceed through theintersection.’ The inference system may also add additional limitationsto various actions for safer operation. Such additional limitationscould include redundant sensor data checks (so the vehicle does not movebased on a sensor glitch), heterogeneous sensor checks (e.g., radar andoptical sensor inputs should agree), additional wait times, etc. Theadditional limitations may be added to provide more reliable and safeoperation of the vehicle while in the autonomous mode.

In some scenarios, the inference system could request vehicledriver/occupant input to confirm or reject an inference. For example,when approaching an intersection where the traffic signal is flashingred, the inference system could infer that the traffic signal isnon-functional and may ask for input from an occupant of the vehicle.The inference system could ask, “This traffic light appears broken, canyou please confirm?” The vehicle occupant could respond, “Yes, it looksbroken,” or “No, I think it normally flashes red.” Such input from thevehicle occupant could be used alone or in combination with sensor datato make further inferences about the aspect of the environment and/or tocontrol the vehicle. The interaction between the inference system andthe vehicle occupant could be carried out using voice, text, or otherforms of communication.

In another example embodiment, while approaching an intersection, atraffic signal could be occluded from the vehicle sensor system. Thevehicle may come to a stop at the intersection. After waiting at leastone cycle time of the traffic light and failing to receive otherappropriate sensor data to proceed through the intersection, theinference system could ask the vehicle occupant “Is it safe to proceedthrough this intersection?” The vehicle occupant could respond with “Go”or “Don't Go”. The vehicle could be controlled based on the responsefrom the vehicle occupant. In other embodiments, the vehicle occupantcould take manual control of the vehicle to negotiate the intersection.Other interactions between the inference system and the vehicle occupantare possible.

In some embodiments, inferences could be transmitted to one or moreother vehicles configured to operate in an autonomous mode. Forinstance, an inference about a state of a traffic signal could betransmitted to vehicles behind the inferring vehicle as an ‘earlywarning’ to avoid abrupt braking or other emergency conditions.

Inferences could also be transmitted in the form of a query to nearbyautonomous vehicles that may be operable to support or disprove theinference. For example, a vehicle stopped behind a large truck thatcould be blocking a traffic signal may infer that the large truck isstopped due to the state of the traffic signal being “red”. In someembodiments, the vehicle may broadcast the inference in an effort toobtain further information about the state of the traffic signal. Inresponse, other vehicles may respond with other information to supportor disprove the inference.

Based on the inference made by the inference system, a controlinstruction or a set of control instructions could be generated by theinference system and transmitted or otherwise communicated to thevehicle or another computer system operable to control part or all ofthe functions of the vehicle.

Step 406 includes controlling the vehicle in the autonomous mode basedon the at least one inferred aspect of the environment of the vehicle.Controlling the vehicle could represent the vehicle following thecontrol instruction(s) in order to perform one or more of the vehicleaccelerating, decelerating, changing heading, changing lane, shiftingposition within a lane, providing a warning notification (e.g.,vehicle-to-vehicle communication message, horn signal, light signal,etc.), and changing to a different driving mode (e.g., semi-automatic ormanual mode). Other actions involving controlling the vehicle arepossible within the context of the present disclosure.

The vehicle could be controlled based at least partially on theconfidence level of the inference or predetermined scenario.

The vehicle and any on-board computer systems could control the vehiclein the autonomous mode. Alternatively, computer systems, such as aserver network, could be used to control some or all of the functions ofthe vehicle in the autonomous mode.

The methods and apparatus disclosed herein could be applied to asituation in which the vehicle operating in the autonomous modeapproaches a freeway metering light (e.g., at a highway on-ramp). In theexample situation, while the vehicle is stopped alongside the meteringlight, the field of view of the camera may not be sufficient to directlyobserve the metering light. The vehicle may receive information (e.g.,from a highway administrator or other roadway information service) aboutthe particular metering light. The received information could includewhether metering is on at the particular metering light as well as thecurrent metering interval. The information could be received via awireless communication interface. As such, in situations where thevehicle may not be able to directly observe the metering light, thereare several ways the vehicle could infer a proper interval time and/or astate of the metering light in order to proceed from the metering light.

For example, the vehicle may observe other cars and infer an intervaltime from movements of the other vehicles. In such an example, thevehicle could observe other vehicles advancing at a certain rate (e.g.,approximately one every five seconds). Based on the observation of othervehicles, the vehicle may infer that the interval is approximately fiveseconds and proceed from the metering light after waiting the inferredinterval time.

If the interval information is known (e.g., based on prior historicaldata or received information), the vehicle may infer the proper meteringinterval based on that interval information. For instance, the highwayauthority may provide metering interval information to the vehicle(e.g., the interval at the current metering light is seven seconds).Based on the interval information, the vehicle may infer the interval tobe seven seconds and proceed from the metering light after waiting sevenseconds after a previous vehicle departure. However, if the vehicleobserves other vehicles are waiting much more time (or less time) thanthe interval information indicates, a new interval may be inferred basedon real-time observations.

In a scenario that includes a two-lane metering light, the vehicle mayobserve other vehicles proceed from the metering light in an alternatingfashion (e.g., right lane, left lane, right lane . . . ). Based on thedeparture interval, the vehicle could infer an appropriate time to waitat the metering light. Further, based on the departure interval and thepresent lane of the vehicle, the vehicle could infer the proper order inwhich to depart from the metering light.

In another example embodiment, the vehicle may make an inference aboutthe state of the metering light based on previous observations of themetering light. For instance, as the vehicle is approaching the meteringlight, it may observe the state of the metering light. However, the viewof the metering light may become occluded by another vehicle or themetering light may be outside the camera field of view. In such asituation, the vehicle may infer a current state of the metering lightbased on the previously observed state of the metering light. Inparticular, the vehicle may observe a transition of the metering light(e.g., red to green or green to red) while moving towards the meteringlight, which may then become unobservable. If the interval informationis known to the vehicle, the vehicle could make an inference regardingthe current state of the metering light based on the intervalinformation and the elapsed time since the previously observedtransition.

The vehicle may deliberately approach the metering light slowly whenmetering is on to assure that a whole metering cycle (or at least onemetering light transition) is observed before the vehicle enters alocation where the metering light is unobservable. Thus, the vehicle maybe able to make a more confident inference about the current state ofthe metering light.

In yet another embodiment, the vehicle may approach an unobservablemetering light with no other vehicles waiting at it or proceedingthrough it. If the vehicle has information that metering is on as wellas information about the interval for the metering light, the vehiclemay just wait at the metering light for the duration of the interval.Thus, despite not directly observing the metering light, the vehiclecould make inferences about the metering light and control the vehiclein an effort to be safe and also to be in accordance with the spirit oftraffic laws and regulations.

A method 410 is provided in which sensor data may be obtained from atleast one sensor observing at least one light source in an environmentof the vehicle when a state of a controlling traffic signal is notdirectly observable. Based on the sensor data, an inference system couldinfer an inferred state of the controlling traffic signal. The vehiclemay be controlled based on the inferred state of the controlling trafficsignal. The method could be performed using any of the apparatus shownin FIGS. 1 and 2 and described above, however, other configurationscould be used. FIG. 4B illustrates the steps in an example method,however, it is understood that in other embodiments, the steps mayappear in different order and steps could be added or subtracted.

Step 412 includes obtaining sensor data from one or more sensorsobserving at least one light source in an environment of a vehicle whilea state of a controlling traffic signal is not directly observable usingthe one or more sensors. The vehicle could be configured to operate inan autonomous mode and the one or more sensors may be operationallyassociated with the vehicle.

The one or more sensors could include any type of sensor as describedelsewhere in the present disclosure. The one or more sensors could beattached to the vehicle or could be located elsewhere.

The at least one light source could include one or more brake lights ofanother vehicle. Other light sources are possible.

The controlling traffic signal could represent any traffic-regulatingdevice, such as a traffic light, a stop sign, or a metering light. Forexample, the vehicle could be approaching an intersection. In such acase, the controlling traffic signal could be a traffic light thatregulates traffic flow through the intersection. Other types ofcontrolling traffic signals are possible. In an example embodiment, thecontrolling traffic signal could be obstructed by another vehicle, abuilding, a tree, or any other type of obstruction such that thecontrolling traffic signal is not directly observable using the one ormore sensors of the vehicle. In other embodiments, the controllingtraffic signal could be outside the observable range of the one or moresensors of the vehicle.

Step 414 includes using an inference system to infer an inferred stateof the controlling traffic signal based on the sensor data. In anexample embodiment, the inference system could be a computer systemassociated with the vehicle, such as computer system 112 described inreference to FIG. 1. In other embodiments, the inference system could beassociated in full or in part with another computer system (e.g., aserver network).

As described elsewhere in the present disclosure, the inference systemcould be operable to make inferences based on sensor data. In someembodiments, the inference system could make inferences about anunobservable state of a controlling traffic signal. For example, avehicle may be approaching an intersection with a controlling trafficsignal. In such a case, a truck may prevent one or more sensorsassociated with the vehicle from directly observing a state of thecontrolling traffic signal. The vehicle could acquire sensor datarelated to one or more light sources in the environment of the vehicle,such as brake lights of other vehicles. If the brake lights of othervehicles are on, the inference system could infer that the state of thecontrolling traffic signal is red or ‘stop’. If the brake lights of theother vehicles are not on, the inference system could infer that thestate of the controlling traffic signal is green or ‘go’. Other types ofinferences are possible.

Step 416 includes controlling the vehicle in the autonomous mode basedon the inferred state of the controlling traffic signal. In an exampleembodiment, if the inferred state of the controlling traffic signal isgreen or ‘go’, the vehicle could be controlled to ‘go throughintersection’ or ‘proceed with caution’. If the inferred state of thecontrolling traffic signal is red or ‘stop’, the vehicle could becontrolled to stop. In some embodiments, the vehicle may be controlledto alert the driver and/or to enter a semiautomatic or manual mode.Other ways of controlling the vehicle in the autonomous mode based onthe inferred state of the controlling traffic signal are possible.

Example methods, such as method 400 of FIG. 4A and method 410 of FIG.4B, may be carried out in whole or in part by the vehicle and itssubsystems. Accordingly, example methods could be described by way ofexample herein as being implemented by the vehicle. However, it shouldbe understood that an example method may be implemented in whole or inpart by other computing devices. For example, an example method may beimplemented in whole or in part by a server system, which receives datafrom a device such as those associated with the vehicle. Other examplesof computing devices or combinations of computing devices that canimplement an example method are possible.

In some embodiments, the disclosed methods may be implemented ascomputer program instructions encoded on a non-transitorycomputer-readable storage media in a machine-readable format, or onother non-transitory media or articles of manufacture. FIG. 5 is aschematic illustrating a conceptual partial view of an example computerprogram product that includes a computer program for executing acomputer process on a computer system, arranged according to at leastsome embodiments presented herein.

In one embodiment, the example computer program product 500 is providedusing a signal bearing medium 502. The signal bearing medium 502 mayinclude one or more programming instructions 504 that, when executed byone or more processors may provide functionality or portions of thefunctionality described above with respect to FIGS. 1-4. In someexamples, the signal bearing medium 502 may encompass acomputer-readable medium 506, such as, but not limited to, a hard diskdrive, a Compact Disc (CD), a Digital Video Disk (DVD), a digital tape,memory, etc. In some implementations, the signal bearing medium 502 mayencompass a computer recordable medium 508, such as, but not limited to,memory, read/write (R/W) CDs, R/W DVDs, etc. In some implementations,the signal bearing medium 502 may encompass a communications medium 510,such as, but not limited to, a digital and/or an analog communicationmedium (e.g., a fiber optic cable, a waveguide, a wired communicationslink, a wireless communication link, etc.). Thus, for example, thesignal bearing medium 502 may be conveyed by a wireless form of thecommunications medium 510.

The one or more programming instructions 504 may be, for example,computer executable and/or logic implemented instructions. In someexamples, a computing device such as the computer system 112 of FIG. 1may be configured to provide various operations, functions, or actionsin response to the programming instructions 504 conveyed to the computersystem 112 by one or more of the computer readable medium 506, thecomputer recordable medium 508, and/or the communications medium 510.

The non-transitory computer readable medium could also be distributedamong multiple data storage elements, which could be remotely locatedfrom each other. The computing device that executes some or all of thestored instructions could be a vehicle, such as the vehicle 200illustrated in FIG. 2. Alternatively, the computing device that executessome or all of the stored instructions could be another computingdevice, such as a server.

The above detailed description describes various features and functionsof the disclosed systems, devices, and methods with reference to theaccompanying figures. While various aspects and embodiments have beendisclosed herein, other aspects and embodiments are possible. Thevarious aspects and embodiments disclosed herein are for purposes ofillustration and are not intended to be limiting, with the true scopeand spirit being indicated by the following claims.

1-28. (canceled)
 29. A method, comprising: obtaining, using one or moresensors, first sensor data regarding an environment of the vehicle andsecond sensor data regarding the environment of the vehicle, wherein theone or more sensors are operationally associated with a vehicleconfigured to operate in an autonomous mode; making a determinationwhether the first sensor data and the second sensor data agree regardingan aspect of the environment of the vehicle; and controlling the vehiclein the autonomous mode based on the determination.
 30. The method ofclaim 29, wherein the first sensor data is image data regarding theaspect of the environment of the vehicle, and wherein the second sensordata is data regarding one or more other aspects of the environment ofthe vehicle.
 31. The method of claim 30, wherein making thedetermination whether the first sensor data and the second sensor dataagree regarding the aspect of the environment of the vehicle, comprises:inferring the aspect of the environment of the vehicle based on thesecond sensor data; and determining whether the first sensor dataregarding the aspect of the environment of the vehicle agrees with theinferred aspect of the environment of the vehicle.
 32. The method ofclaim 31, wherein the aspect of the environment is a state of a trafficsignal determined from the first sensor data, wherein the one or moreother aspects of the environment of the vehicle comprise a status of atleast one other vehicle in the environment determined from the secondsensor data, and wherein the inferred aspect of the environment of thevehicle is an inferred state of the traffic signal based on the statusof the at least one other vehicle in the environment.
 33. The method ofclaim 32, wherein the traffic signal is at an intersection of a firstroadway and a second roadway, wherein the vehicle is on the firstroadway, and wherein controlling the vehicle in the autonomous modebased on the determination comprises: if the state of the traffic signaldetermined from the first sensor data is green and the inferred state ofthe traffic signal based on the status of the at least one other vehicledetermined from the second sensor data is green, controlling the vehicleto proceed through the intersection; and if the state of the trafficsignal determined from the first sensor data is green but the inferredstate of the traffic signal based on the status of the at least oneother vehicle determined from the second sensor data is not green,controlling the vehicle to stop at the intersection.
 34. The method ofclaim 32, wherein the status of the at least one other vehicle in theenvironment comprises at least one vehicle on the first roadway that isstopped at the intersection, and wherein the inferred state of thetraffic signal is red based at least in part on the at least one vehicleon the first roadway that is stopped at the intersection.
 35. The methodof claim 32, wherein the status of the at least one other vehicle in theenvironment comprises at least one vehicle on the second roadwayproceeding through the intersection, and wherein the inferred state ofthe traffic signal is red based at least in part on the at least onevehicle on the second roadway proceeding through the intersection. 36.The method of claim 29, wherein the one or more sensors includes acamera and at least one other sensor, wherein the first sensor data isobtained using the camera and the second sensor data is obtained usingthe at least one other sensor.
 37. A non-transitory computer readablemedium having stored therein instructions executable by a computersystem to cause the computer system to perform functions, the functionscomprising: obtaining, using one or more sensors, first sensor dataregarding an environment of the vehicle and second sensor data regardingthe environment of the vehicle, wherein the one or more sensors areoperationally associated with a vehicle configured to operate in anautonomous mode; making a determination whether the first sensor dataand the second sensor data agree regarding an aspect of the environmentof the vehicle; and controlling the vehicle in the autonomous mode basedon the determination.
 38. The non-transitory computer readable medium ofclaim 37, wherein the first sensor data is image data regarding theaspect of the environment of the vehicle, and wherein the second sensordata is data regarding one or more other aspects of the environment ofthe vehicle.
 39. The non-transitory computer readable medium of claim38, wherein making the determination whether the first sensor data andthe second sensor data agree regarding the aspect of the environment ofthe vehicle, comprises: inferring the aspect of the environment of thevehicle based on the second sensor data; and determining whether thefirst sensor data regarding the aspect of the environment of the vehicleagrees with the inferred aspect of the environment of the vehicle. 40.The non-transitory computer readable medium of claim 39, wherein theaspect of the environment is a state of a traffic signal determined fromthe first sensor data, wherein the one or more other aspects of theenvironment of the vehicle comprise a status of at least one othervehicle in the environment determined from the second sensor data, andwherein the inferred aspect of the environment of the vehicle is aninferred state of the traffic signal based on the status of the at leastone other vehicle in the environment.
 41. The non-transitory computerreadable medium of claim 40, wherein the traffic signal is at anintersection of a first roadway and a second roadway, wherein thevehicle is on the first roadway, and wherein controlling the vehicle inthe autonomous mode based on the determination comprises: if the stateof the traffic signal determined from the first sensor data is green andthe inferred state of the traffic signal based on the status of the atleast one other vehicle determined from the second sensor data is green,controlling the vehicle to proceed through the intersection; and if thestate of the traffic signal determined from the first sensor data isgreen but the inferred state of the traffic signal based on the statusof the at least one other vehicle determined from the second sensor datais not green, controlling the vehicle to stop at the intersection. 42.The non-transitory computer readable medium of claim 40, wherein thestatus of the at least one other vehicle in the environment comprises atleast one vehicle on the first roadway that is stopped at theintersection, and wherein the inferred state of the traffic signal isred based at least in part on the at least one vehicle on the firstroadway that is stopped at the intersection.
 43. The non-transitorycomputer readable medium of claim 40, wherein the status of the at leastone other vehicle in the environment comprises at least one vehicle onthe second roadway proceeding through the intersection, and wherein theinferred state of the traffic signal is red based at least in part onthe at least one vehicle on the second roadway proceeding through theintersection.
 44. The non-transitory computer readable medium of claim37, wherein the one or more sensors includes a camera and at least oneother sensor, wherein the first sensor data is obtained using the cameraand the second sensor data is obtained using the at least one othersensor.
 45. A vehicle, comprising: one or more sensors, wherein the oneor more sensors are configured to acquire sensor data regarding anenvironment of the vehicle, wherein the vehicle is configured to operatein an autonomous mode; and a control system, wherein the control systemis configured to: use the one or more sensors to obtain first sensordata regarding the environment of the vehicle and second sensor dataregarding the environment of the vehicle; make a determination whetherthe first sensor data and the second sensor data agree regarding anaspect of the environment of the vehicle; and control the vehicle in theautonomous mode based on the determination.
 46. The vehicle of claim 45,wherein the first sensor data is image data regarding the aspect of theenvironment of the vehicle, and wherein the second sensor data is dataregarding one or more other aspects of the environment of the vehicle.47. The vehicle of claim 46, wherein the determination comprises:inferring the aspect of the environment of the vehicle based on thesecond sensor data; and determining whether the first sensor dataregarding the aspect of the environment of the vehicle agrees with theinferred aspect of the environment of the vehicle.
 48. The vehicle ofclaim 47, wherein the aspect of the environment is a state of a trafficsignal determined from the first sensor data, wherein the one or moreother aspects of the environment of the vehicle comprise a status of atleast one other vehicle in the environment determined from the secondsensor data, and wherein the inferred aspect of the environment of thevehicle is an inferred state of the traffic signal based on the statusof the at least one other vehicle in the environment.